Apparatus and Methods for Improving Video Quality From a Digital Video Signal Including Replicated Image Frames

ABSTRACT

A method includes receiving a first digital video signal having frames at a first frame rate. A respective frame of the first digital video signal is then replicated multiple times to produce a series of replicated frames for inclusion in a second digital video signal have a second frame rate greater that the first frame rate. For at least some of the pixel data defining pixels in at least one of the replicated frames, the method includes randomly modifying the respective pixel data. The random modification of pixel data for a respective pixel in a respective replicated frame results in a modified replicated frame which is included in the second digital video signal.

TECHNICAL FIELD OF THE INVENTION

The invention relates to digital video signal processing systems andparticularly to such systems which receive a video signal stream at afirst frame rate and then replicate image frames from that video signalstream to produce a video signal stream at a higher frame rate. Theinvention encompasses methods for video signal processing, apparatus forvideo signal processing, and video systems.

BACKGROUND OF THE INVENTION

Digital video systems generally include image capturing components whichcreate a digital video signal for a desired scene and signal processingcomponents which perform various format conversions or other processingtasks to ultimately produce an input to a digital video display monitorwhich displays a video image of the desired scene.

Digital video signals are typically made up of a stream of data defininga series of image frames which may by displayed at a video monitorrapidly in succession at a desired frame rate to produce a video image.Each image frame (hereinafter referred to simply as a “frame”) is madeup of a number of pixels with each pixel defining the light intensity ata particular point in the frame. Each pixel in a frame is defined bypixel data which specifies the light intensity in terms of a digitalvalue or set of digital values in the case of color video. For example,each pixel in a digital video frame may be represented by three 8-bitvalues each defining a respective color component for the pixel, such asred, green, and blue. In addition to the color component data, the pixeldata may include additional data defining other characteristics of thepoint in the frame such as transparency for example.

In some cases the digital video signal generated by the image capturingcomponents of a digital video system may be at a lower frame rate thanthe frame rate at which the video display monitor operates. For example,a medical video imaging system may employ a light sensor array thatdetects light over a relatively long period of time (that is, a longintegration period) to generate a respective frame and thus may createframes at a slower frame rate than the frame rate at which frames are tobe displayed by the system display monitor. Such a situation may occurwith cameras operating in fluorescence imaging modes where the imagecapture system is light starved and therefore forced to employ a longintegration period for each frame.

Frame replication is a common solution to address the situation in whicha given digital video signal at a first frame rate must be used to drivea video display monitor at a higher frame rate. Frame replication may beaccomplished by buffering the frames of the lower frame rate videosignal in buffer memory so that the frames may be read at a higher rateto produce the desired higher frame rate signal. In this solution, eachframe of the lower frame rate video signal is replicated as many timesas necessary to produce a video signal at the desired higher frame rate.Take, for example, the case where a given set of image capturingcomponents generate a digital video signal at 15 frames per second(“fps”) while the video signal is to be displayed on a display monitorwhich operates at an update/refresh rate of 60 fps. In this case, eachframe of the lower frame rate digital video signal may be buffered insuitable buffer memory, having a suitable buffer memory architecturesuch as a classical triple buffer architecture for example, and readfour times from that memory to produce a digital video signal having thedesired 60 fps rate. That is, each frame of the lower frame rate digitalvideo signal is replicated four times and each replicated frame isplaced in an output digital video signal having the higher frame rate.In this frame replicating technique, the digital video signal ultimatelysent to the display monitor causes the display monitor to show the exactsame frame (exact down to each respective pixel in the frame) four timesbefore the next frame from the lower frame rate digital video signal canbe displayed on the display monitor.

SUMMARY OF THE INVENTION

A problem with the above-described prior art frame replication techniqueis that it accentuates noise perception in the lower frame rate videosignal. That is, even though the display monitor is being refreshed at asufficiently high rate so that noise present in individual frames mightnot be perceived by an observer, the observer sees any noise present inframes of the original, lower frame rate digital video signal over alonger period of time (four times in the above example of 15 fps captureand 60 fps display). Thus the frame replication used to produce a higherframe rate digital video signal from a lower frame rate signal mayresult in a lower quality video image than would otherwise be possibleat the higher update rate at which the display monitor is capable ofoperating.

It is an object of the invention to provide processing methods andprocessing apparatus which may increase the quality of video in videosystems employing frame replication to increase frame rate.

A method of processing a frame of a digital video signal according toone aspect of the present invention includes randomly modifying thepixel data defining pixels in at least one of the replicated frames in aframe replication arrangement. According to this first aspect of theinvention a first digital video signal having a first frame rate isreceived, and a respective frame of the signal is replicated to producea series of replicated frames. Each replicated frame is defined at leastin part by respective pixel data specifying each pixel included in thatreplicated frame. For at least some of the pixel data defining pixels inat least one of the replicated frames, the method includes randomlymodifying the respective pixel data. The random modification of pixeldata for a given pixel in a respective replicated frame results in amodified replicated frame which is then included in a second digitalvideo signal. This second digital video signal includes the modifiedreplicated frame together with additional frames replicated from framesof the first digital video signal to produce a relatively higher framerate for the second digital video signal.

It should be noted that the term “random” as used in this disclosure andthe accompanying claims encompasses both true randomness andpseudo-randomness. That is, a random modification according to thepresent disclosure and claims may comprise a true random modification orany pseudo-random modification.

As noted above in the background section, noise included in theoriginal, lower frame rate digital video signal, and necessarilyreplicated in the replicated frames, may be perceptible to the viewer ofthe higher frame rate digital video signal. However, methods accordingto this first aspect of the invention have the advantage that the randommodifications to the pixel data defining the various replicated framesincluded in the second digital video signal tend to mask noise includedin the original, lower frame rate digital video signal. Thus the second,higher frame rate digital video signal may produce a video display thatis more pleasing to the viewer.

The modification of pixel data according to this first aspect of theinvention is independent of quantization of the pixel data. Thus therandom modification of pixel data according to the present invention isnot a randomization of quantization errors commonly referred to as“dithering.” The random modification of pixel data for various framesincluded in a digital video signal according to aspects of the presentinvention may be applied in addition to any dithering applied to adigital video signal.

According to another aspect of the present invention, an apparatusincludes a video frame replication device configured to receive a firstdigital video signal having a first frame rate, and to replicate arespective frame of the first digital signal multiple times to produce aseries of replicated frames. The series of replicated frames produced bythis replication device are included in a second digital video signalhaving a second frame rate greater than the first frame rate. Theapparatus further includes a frame modification controller configured torandomly modify the respective pixel data for at least some of thepixels of at least one of the replicated frames. This randommodification of pixel data for pixels in a respective replicated frameresults in a modified replicated frame which is included in the seconddigital video signal to provide the noise masking advantage noted aboveas to the previously described aspect of the invention.

Another aspect of the present invention comprises a video system whichincludes a video frame replication device and frame modificationcontroller similarly to the apparatus described above, and furtherincludes an image sensor assembly and camera control module. The imagesensor assembly in this additional aspect of the invention generates thefirst, lower frame rate digital video signal for processing by thecamera control module. The video frame replication device and framemodification controller are included in the camera control moduleaccording to this aspect of the invention.

Implementations according to any of the above aspects of the inventionmay include a dynamic aspect to the modification of the replicatedframes based on one or more characteristics of the first digital videosignal. For example, an implementation may detect a noise level in thefirst digital video signal, and may select a modification level for therandom modification based on this detected noise level. The randommodification of pixel data may then be performed based on the selectedmodification level, that is, based on one or more parameters of theselected modification level as will be described further below. In someimplementations, the noise level detection may be specific to aparticular color component of the first digital video signal. In theseimplementations, the random modification of pixel data may be performedbased on the selected modification level as to the particular colorcomponent, but at different levels for the other color components. Thenoise detection may also be for only one or more sub-areas of a frame orframes making up the first digital video signal (that is, not the entirearea defined by the respective frame), and the modification may only beapplied based on the detected noise level to pixel data for pixels incorresponding sub-areas of the replicated frames. Alternatively, thenoise detection in one or more sub-areas of a frame or frames of thefirst digital video signal may be applied to set the level ofmodification applied to pixel data for pixels over the entire area of areplicated frame.

Some implementations according to the various aspects of the inventionmay include dynamic modification of replicated frames which is not basedon any characteristic of the first digital video signal. For example, animplementation according to any of the above aspects of the inventionmay apply the pixel data modification from one replicated frame to thenext based on some pattern, such as every nth replicated frame where nis an integer greater than one and may or may not be a static value.Regardless of whether a dynamic modification of pixel data is based onsome characteristic of the first digital video signal or not, theseimplementations may include a determination for each given replicatedframe whether the frame is to be modified or left unmodified.

Embodiments of the present invention may include many variations in themanner in which the pixel data is modified. In some cases, the pixeldata for each pixel in a given frame is randomly modified. In othercases, the pixel data for only some of the pixels in a given frame maybe modified. The determination of whether pixel data for a given pixelis modified or not may be random or according to some pattern. In thelatter case, every other pixel or every nth pixel in a frame (where n isan integer greater than 1) may have the corresponding data modified.Furthermore a determination as to the particular modification to beapplied to the pixel data for a given pixel may be random, based on oneor more characteristics of the first digital video signal, or based onsome predetermined pattern.

These and other advantages and features of the invention will beapparent from the following description of illustrative embodiments,considered along with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of a digital video system embodyingaspects of the present invention.

FIG. 2 is a flow diagram showing process steps according to a firstembodiment of the present invention.

FIG. 3 is a flow diagram showing process steps according to anadditional embodiment the present invention.

DESCRIPTION OF REPRESENTATIVE EMBODIMENTS

In the following description, FIG. 1 will first be used to describe adigital video system and apparatus according to aspects of the presentinvention. Methods according to additional aspects of the invention willthen be described with reference to FIGS. 2 and 3 generally and in thecontext of the system shown for example in FIG. 1.

The present invention may be implemented in any digital video system inwhich frames from a first, relatively lower frame rate digital videosignal are replicated and placed in a second digital video signal havinga relatively higher frame rate. Thus the present invention may haveapplication in endoscopic systems, laparoscopic systems, digitalmicroscopes, digital cameras, and mobile phones equipped with imagingsub-systems, for example. While aspects of the invention may bedescribed in terms of the video system shown in FIG. 1, which has aconfiguration commonly found in endoscopes and laparoscopes, it shouldbe appreciated that the invention is by no means limited to this exampleconfiguration.

The digital video system 100 illustrated in FIG. 1 includes threesubsystems, an image sensor assembly is shown as dashed box 101, acamera control unit shown as dashed box 102, and a user interface shownas dashed box 103. In this example embodiment, image sensor assembly 101includes elements for sensing an image and producing a digital videosignal from the sensor output. Camera control unit 102 includes elementsfor processing the digital video signal received from image sensorassembly 101, and for placing the signal in a form which may be used todrive an image display monitor included in user interface 103. Thisfunctionality includes frame replication for the digital video signalreceived from image sensor assembly 101 and frame modification accordingto the present invention as will be described in detail below. Theillustrated camera control unit 102 also generates control signals forboth image sensor assembly 101 and associated elements and receivescontrol inputs from elements of user interface 103. User interface 103in this example video system includes elements for displaying imagesgenerated from the digital video signal output from camera control unit102, for providing an operator display, and for receiving user inputswhich may be used to operate and control the video system.

A light source 108 directs light to illuminate a subject scene shown inFIG. 1 at reference number 109 so that image sensor assembly 101 mayproduce a useable digital video image of the subject scene. Inparticular, light source 108 illuminates subject scene 109 with visiblelight and/or fluorescent excitation light which may be outside thevisible spectrum in the ultra-violet range or the infra-red/nearinfrared range, or both. Light source 108 may include a single lightemitting element configured to provide light throughout the desiredspectrum, or multiple light emitting elements for producing the desiredlight for illuminating subject scene 109. In some implementations, lightsource 108 may include light emitting devices such as LEDs which areplaced proximate to subject scene 109, while in other implementationslight emitting devices may be placed remotely and the emitted light maybe directed to the subject scene through a suitable light guide such asone or more optical fibers. For example, in medical endoscopicapplications, both light source 108 and image sensor assembly 101 may beincluded in an endoscope connected to camera control unit 102. In theseimplementations, light source 108 may be located at the tip of theendoscope shaft or may be located in a body or camera head for theendoscope. In the former case light source 108 directly illuminates asubject scene proximate to the endoscope tip, while in the latter caselight from the light source is directed to the subject scene at theendoscope tip through a light guide arrangement extending through theendoscope shaft. Regardless of the nature of light source 108, theemitted light results in light 110 from subject scene 109 which iscollected and focused by one or more lenses included in an opticalassembly 111 to form an image at image sensor 120 included in imagesensor assembly 101. It will be appreciated that light 110 may be lightfrom light source 108 which is reflected from the subject scene 109 or,as in the case of fluorescence imaging, may be light emitted from thesubject scene in response to excitation light incident on the subjectscene from the light source.

The example embodiment of FIG. 1 is illustrated as including a singleimage sensor as part of image sensor assembly 101. It will beappreciated, however, that a given sensor 120 may comprise an array ofdiscrete image sensing elements such as separate R, G, and B sensorarrays. Also, video systems according to the present invention mayinclude multiple types of sensors rather than a single sensor type. Forexample, an image sensor assembly within the scope of the presentinvention may include multiple different types of sensor elements eachoperable to sense light in a different spectrum. In any case, sensor 120or other light sensors included in a video system embodying principlesof the present invention may be active pixel complementary metal oxidesemiconductor sensor devices (CMOS APS), charge-coupled devices (CCD),or any other digital imaging sensor devices now known or developed inthe future. Regardless of the particular type or configuration of imagesensor 120, the element converts the incident within its operatingspectrum to an electrical signal by integrating charge for each pixeldefining a point in the image.

The total amount of light 110 reaching the image sensor 120 may beregulated by the intensity of light emitted from light source 108 and anaperture (not shown separately) associated with optical assembly 111.Control of light source 108 and optical assembly 111 may be provided bycontrol signals from camera control unit 102 as indicated by the signalpaths shown in FIG. 1 from the camera control unit to the light sourceand optical assembly. Camera control unit 102 may also directly orindirectly control the time over which image sensor 120 integratescharge to capture a frame of subject scene 109 to be included in thedesired video signal. The random modification of frames according to thepresent invention has particular application in situations in which thecharge integration period for each frame is relatively long and thusframes may be produced at a relatively low rate, at least lower than theframe rate at which the desired image may ultimately be displayed in thevideo system. These long charge integration and low image frame ratesituations are examples of situations in which frame replication may beemployed as will be described further below, and in which randommodification of replicated frames according to the present invention maybe desirable.

The analog signals produced by image sensor 120 are processed by analogsignal processor 122 in the example implementation of FIG. 1, and thendigitized at analog-to-digital (A/D) converter 124 to produce a streamof frame data containing the various color channels employed in the caseor color video. Timing generator 126 produces various clocking signalsto select rows and pixels and synchronizes the operation of image sensor120, analog signal processor 122, and A/D converter 124 to produce thedesired stream of frame data. This stream of frame data output from A/Dconverter 124 may represent a first digital video signal in the contextof the present invention, and is directed to camera control unit 102 forfurther processing.

The example camera control unit 102 shown in FIG. 1 includes a systemcontroller 130 along with system memory 136 and program memory 138. Theexample camera control unit 102 also includes image processingcontroller 140, a video encoder 150, and a display controller 152. Thisparticular implementation further includes a frame modificationcontroller 160, and a video frame replication device 162 interposedbetween system controller 130 and video encoder 150. Video framereplication device 162 in this illustrated embodiment may comprise abuffer arrangement that replicates frames from the first digital videosignal received from image sensor assembly 101 as necessary to produce adigital video signal having the desired frame rate, while framemodification controller 160 randomly modifies replicated framesaccording to aspects of the present invention as will be discussed infurther detail in connection with the process flow diagrams of FIGS. 2and 3.

The system controller 130 controls the overall operation of the videosystem 100 based on a software program stored in program memory 138.This memory can also be used to store user setting selections and otherdata to be preserved when the system is turned off. In particular,system controller 130 controls the data capture which results in thefirst digital video signal by setting light source 108 intensity, theoptical assembly 111 aperture, and any filters included in opticalassembly 111, and by controlling the timing that may be necessary toobtain the image stream based on the sensed light.

Image processing controller 140 may function to provide any number ofimage processing tasks in video system 100. In implementations in whichthe modification of frames according to the present invention isdependent upon a detected noise level in the first digital video signal,the noise detection may be performed by image processing controller 140.In these implementations image processing controller 140 may also beconfigured to select a modification level to be applied by framemodification controller 160. Where included in the system, the noisedetection may be performed in any suitable fashion. Image processingcontroller 140 may also perform operations on the incoming digital videosignal for purposes other than in connection with frame modificationaccording to the present invention. For example, when video system 100comprises a medical endoscopic system, image processing controller maymodify the data of the incoming digital video signal to accentuateaspects of the video signal data to enhance the diagnostic value of theimages.

Image data including a stream of frames that have been processedaccording to image processing controller 140, replicated to provide adesired frame rate by video frame replication device 162, and modifiedaccording to the present invention by frame modification controller 160are continuously sent to video encoder 150. As is well known in the art,video encoder 150 places the incoming image data in a desired compressedformat for display controller 152. Display controller 152 then generatesthe desired video signal (HDMI or DVI, for example) which provides aninput to image display monitor 170 of user interface 103. Image displaymonitor 170 functions to display video images according to the inputsignal from camera module 102, and may comprise any suitable displaydevice such as a liquid crystal display backlit with light-emittingdiodes (LED LCD), for example.

In addition to image display monitor 170, user interface 103 in theexample video system shown in FIG. 1 includes a user input arrangement172 and an operator display 174. User input arrangement 172 may includea keyboard, computer pointing devices, buttons, rocker switches,joysticks, rotary dials, or touch screens (implemented on operatordisplay 174 or some other display of the user interface 103). Operatordisplay 174 may include any suitable arrangement for providing systemstatus or operation information to an operator of the system, andpreferably includes a display device such as an LED LCD display throughwhich status and operating information may be displayed to the systemoperator. Both user input arrangement 172 and operator display 174communicate with system controller 130 over suitable communicationlinks. For example, devices included in user input arrangement 172 maycommunicate with system controller 130 over a suitable serialcommunication connection such as USB or RS-232. Operator display 174 maycommunicate with system controller over a suitable video cable.

The functional block diagram of FIG. 1 is provided to illustrate anexample video system in which frame replication and frame modificationmay be implemented according to various aspects of the presentinvention. Not only may the present invention be employed in any framereplication context, but also numerous variation are possible in a videosystem such as that shown in FIG. 1. For example, although thefunctional block diagram of FIG. 1 shows the functional elements ofimage sensor assembly 101 as discrete elements, these functionalelements or various combinations of these elements may be fabricated asa single integrated circuit as is commonly done with CMOS image sensors.Similarly, although the illustrated distribution of functional elementsof camera control module 102 among multiple programmable logic devices,processors, and controllers is typical, these programmable logicdevices, processors, or controllers can be combinable in various wayswithout affecting the functional operation of the video system and theapplication of the invention. These programmable logic devices,processors, or controllers can comprise one or more programmable logicdevices, digital signal processor devices, microcontrollers, or otherdigital logic circuits. Although a combination of such programmablelogic devices, processors, or controllers has been described, it shouldbe apparent that one programmable logic device, digital signalprocessor, microcontroller, or other digital logic circuit may bedesignated and programmed or otherwise configured to perform all of theneeded functions described above for camera control module 102. All ofthese variations can fall within the scope of the present invention.

Methods of processing digital video signal frames may now be describedwith reference to FIGS. 2 and 3. While the methods shown in FIGS. 2 and3 will be described in some cases below in the context of the examplesystem shown in FIG. 1, it should be borne in mind that methodsaccording to the invention are certainly not limited to implementationvia the system shown in FIG. 1, and, as noted above, may be applied inany digital video system or apparatus that replicates frames from onedigital video signal having a first frame rate to produce a seconddigital video signal having a second, higher frame rate.

Referring now to FIG. 2, a process according to one implementation ofthe present invention includes receiving a first digital video signal asshown at process block 200. The process further includes replicatingeach frame of the first digital video signal as indicated at processblock 202 so as to create a series of replicated frames. As shown atprocess block 204, the example method of FIG. 1 further includesrandomly modifying the pixel data of each replicated frame. Eachmodified replicated frame is then included in a second digital videosignal according to process block 206. In the illustration of FIG. 2,the process steps 202, 204, and 206 may be repeated for each frame ofthe received first digital video signal to produce the desired seconddigital video signal.

In the context of the video system shown in FIG. 1, the digital videosignal generated by image sensor assembly 101 and communicated to cameracontrol unit 102 may represent the first digital video signal receivedas shown at process block 200 in FIG. 2. This first digital video signalmay be in any format, however, regardless of format the signal defines astream of frames, each frame represented by an orthogonal bit map ofpixels, with each pixel defined by a digital value and representingpixel data for the respective pixel. In a video system such as thatshown in FIG. 1, the first digital video signal communicated from sensorassembly 101 to camera control unit 102 may comprise an uncompresseddigital video signal. However, it is possible that the first digitalvideo signal in an embodiment of the present invention may be receivedin some compressed form in which each frame is represented in somecompressed form of data. In such a case the signal may be decompressedto facilitate processing, including frame modification according to thepresent invention. It is also possible that frame modification accordingto the present invention may be performed on a compressed digital videosignal.

The video frame replicating process shown at process block 202 in FIG. 2may be performed, for example, in any suitable data bufferingarrangement such as a triple buffer in which each frame of the firstdigital video signal is buffered into memory and simultaneously readmultiple times to produce the desired series of replicated frames.Regardless of the particular hardware and control employed, the framereplication allows frames of the first digital video signal to bereplicated as many times as necessary to produce a stream of frames at adesired frame rate higher than the frame rate of the first digital videosignal. This frame replication is performed in the example system ofFIG. 1 by video frame replication device 162. Although the functionalblock diagram of FIG. 1 indicates that the frame replication isperformed immediately prior to frame modification, other implementationsmay include additional processing steps between frame replication asshown at process block 202 in FIG. 2 and frame modification as shown atprocess block 204. It is also possible that the frame modification shownas a separate block 204 in FIG. 2 is incorporated into the framereplication. In other words, the replication shown at process block 202may not replicate frames exactly, but may instead replicate the frameswith random modifications according to process block 204.

As will be described below in connection with FIG. 3, other methodswithin the scope of the present invention include modifying onlyselected frames in the stream of replicated frames and leaving someframes unmodified. However, in the example process shown in FIG. 2 therandom modification of pixel data is applied to each replicated frame.This pixel data modification shown at process block 204 in FIG. 2 isperformed by frame modification controller 160 in the video system shownin FIG. 1. Even in implementations such as that shown in FIG. 2, themodification of pixel data is subject to wide variation within the scopeof the present invention. In some implementations, the pixel data foreach pixel in the given frame is randomly modified, while in otherimplementations the pixel data for every other pixel, or more generallyever nth pixel (where n is a constant or varying integer greater than 1)may be randomly modified. Also, the random modification may be appliedon a line-by-line basis in the array of pixels making up a given frame,with the same random modification being applied to each pixel in a givenline or given number of lines of pixels in the frame. The randommodification may comprise a random addition to or subtraction of “1”from each color component value included in the pixel data.Alternatively, the value added to or subtracted from each colorcomponent value for a given pixel may be a binary value selectedrandomly from within a range of values. For example, for each pixel tobe modified, a modification may be selected from a set of values 0through 4 or more. Thus each color component within the pixel data for agiven pixel may be left unchanged or increased or decreased by values of1, 2, 3, 4, or more. In yet other alternatives for the modification ofpixel data shown at process block 204 in FIG. 2, the modification maynot be applied to each color component of the pixel data for a givenpixel. Rather, only a single or some other subset of the colorcomponents may be randomly modified. The number of color components tobe modified for a given pixel may also be randomly determined within thescope of the present invention.

Regardless of the particular random modification applied to a givenframe as shown at process block 204 in FIG. 2, the modified frame isultimately included in a stream of frames making up the second digitalvideo signal as shown at process block 206 in FIG. 2. This step ofincluding the given modified frame in the second digital video signalmay be performed in any suitable manner within the scope of the presentinvention. In some embodiments the output of the frame replicationdevice comprising a stream of frames made up of succeeding series ofreplicated frames may comprise the second digital video signal, and themodification performed as shown at process block 204 in FIG. 2 resultsessentially in a modified frame included in the second digital videosignal. In other implementations, a separate component before or afterthe component which modifies the frames may assemble the replicatedmodified frames as desired into a stream of frame data comprising thesecond digital video signal.

The process shown in FIG. 3 is similar to that shown in FIG. 2 in thatit includes receiving a first digital video signal as shown at processblock 300, replicating a respective frame from the first digital videosignal as shown at process block 304, modifying pixel data for areplicated frame at process block 312, and including the modifiedreplicated frame in a second digital video signal as shown at processblock 314. These for process steps, 300, 304, 312, and 314, may be asdescribed above for the corresponding steps 200, 202, 204, and 206 ofFIG. 2. However, the process shown in FIG. 3 includes additional processsteps to support dynamic (for example, time varying) modification ofreplicated frames, including dynamic modification based oncharacteristics of the first digital video signal and dynamicmodification not based on any characteristic of the first digital videosignal. In particular, the process shown in FIG. 3 includes analyzingthe first digital video signal as shown at process block 302,determining if a given replicated frame in the series of replicatedframes is to be modified as shown at process block 306, and dependingupon the outcome of the decision indicated at decision box 308, eitherincluding the given frame in the second digital video signal unmodifiedor determining a modification to be applied as shown at process block310. The steps from process block 304 to 314 may be performed on aframe-by-frame basis and thus repeated for each frame of the incomingfirst digital video signal and for each replicated frame producedaccording to process block 304.

The analysis indicated at process block 302 in FIG. 3 may comprise ananalysis of one or more frames of the first digital video signal todetect some characteristic of the frame that may be used to affect theframe modification performed at process block 312 in FIG. 3. Forexample, the analysis at process block 302 may be to determine a noiselevel in the first digital video signal. Such a noise level may be usedat process block 310 in FIG. 3 to determine the modification to beapplied to a given frame. In particular, a noise level detected for aframe of the first digital video signal at process block 302 may be usedat process block 310 to select a level of modification for framesreplicated from that frame or other frames in the first digital videosignal. Continuing with this example, the determination at process block310 may include selecting a random modification of pixel data at thesame or similar level as the detected noise level. The noise leveldetected at the analysis indicated at process block 302 may be a noiselevel as to all aspects of the frame or a noise level in a single orless than all color components of the frame. In the latter case, thedetermination at process block 310 may include selecting a randommodification level only as to the color component or components includedin the detection at process block 302. In any event, the analysis may belimited to a subset of pixels of the given frame to simplify theanalysis and the detected characteristic may be used to select amodification level as to just that subset of pixels or the entire set ofpixels making up the frame or some other subset of such pixels.Similarly, the analysis at process block 302 may not be conducted foreach image frame included in the first digital video signal but for somesample number of frames periodically analyzed for the desiredcharacteristic to be detected.

A noise level detected in accordance with the analysis shown at 302 inFIG. 3 may, for example, indicate a noise level in terms of a number ofdigital levels in one or more of the digital values included in thepixel data for a given pixel or given set of pixels for a frame.Continuing with this example, a noise level detected at process block302 may indicate noise at plus or minus 1, 2, 3, 4, or more digitalvalues. In such a case, the modification selected at process block 310may be a modification or an average modification applied at random ofthe same digital value detected at block 302. In a specific case, thenoise level detected may be at plus or minus 3 digital values and themodification selected at process block 310 may also be at 3 digitalvalues or selected to average plus or minus 3 digital values. Of coursethere need not be an exact correspondence between a noise level detectedat process block 302 and a modification selected at process block 310,although there may be some relationship between the detected noise leveland selected modification level such that the modification is based atleast partially on the detected noise level.

The analysis indicated at process block 302 may also affect thedetermination at process block 306 in FIG. 3. For example, a processaccording to the present invention may include dynamically determiningwhether frame modification is to be employed for a given frame or framesreplicated from the first digital video signal. In one example, wherethe analysis at 302 in FIG. 3 indicates a noise level in the firstdigital video signal below some predefined threshold, the determinationat process block 306 may be to forego any modification of a given frameas being unnecessary to improve the second, higher frame rate videosignal.

The determinations at process blocks 306 and 310 may also includeaspects which are not dependent upon any characteristic of the firstdigital video signal analyzed as indicated at process block 302 in FIG.3. For example, an implementation of the present invention may beconfigured to modify every other frame or every nth replicated frame inthe series of replicated frames, or in a series of m replicated frames.In these implementations the determination at process block 306 may be adetermination as to whether the given frame fits the applicabledefinition of a frame that is to be modified, i.e., whether the givenframe is the nth frame from the last modified frame. In otherimplementations, the determination made at process block 306 may be arandom determination to select a given frame to be modified,particularly a random selection to ensure a certain predefined averagenumber of frames to be modified in a given stream.

At process block 310 in FIG. 3, the determination as to a modificationto be applied to a given frame may be based on many factors unrelated toany characteristic of the first digital video signal. For example, thedetermination at process block 310 may be a random determination or arandom determination weighted to ensure an overall proportion of framesare modified over a given number of frames. Continuing with thisexample, there may be multiple predefined levels of modification to beapplied to a given frame, each perhaps defining which pixel data tomodify and to what extent. The determination at process block 310 maythen be to select one of these predefined levels of modification. Whereapplied, any of these determinations as to the modification to beapplied may be performed at a frame modification controller such as thatshown as element 160 in the embodiment of FIG. 1. Alternatively, therandom modifications to be performed may be selected by a separateprocessing device such as that shown at system controller 130 in FIG. 1,and such a separate processing device may control the processing elementthat actually applies the frame modification. Furthermore, selecting themodification to apply to a given frame in accordance with process block310 in FIG. 3 need not be done immediately prior to the givenmodification. Rather, the modification applied at process block 312 inFIG. 3 may be performed according to a script that is generated for agiven stream of frames or applied generally to any stream of frames, andthus a predefined script may control the determination indicated atprocess block 310 in FIG. 3.

Implementations within the scope of the present invention may includenumerous variations on the representative process shown for in FIG. 3.For example, the determinations at process block 306 and 310 need not beperformed for each replicated frame as suggested by FIG. 3. Oneimplementation along this line may perform the determinations at one orboth blocks 306 and 310 only for the first replicated frame of a seriesof replicated frames, and then the determinations as to that frame maybe applied to each replicated frame in the series. Other implementationsmay omit either one of the determinations shown at process blocks 306and 310, or may omit the analysis indicated at process block 302 andinclude one or both of the determinations at process blocks 306 and 310.

It should be noted that frame modification according to the variousaspects of the present invention is not limited to any particular framereplication scheme. While the frame replication shown for example atprocess block 202 in FIGS. 2 and 304 in FIG. 3 may include replicatingeach frame in the first digital video signal to the same extent, someimplementations may apply other frame replication schemes. For example,the frame replication shown at process block 202 in FIG. 2 or processblock 304 in FIG. 3 may replicate every other frame of the first digitalvideo signal, or more generally, every xth frame of a series of y framesof the first digital video signal. It is also possible that a framereplication scheme may replicate some frames of the first digital videosignal to a first extent, and other frames of the first digital videosignal to a second different extent. Replicated frame modificationaccording to aspects of the present invention may be applied to anyreplicated frame.

Also, although noise level is provided above as an example of acharacteristic that may be used to affect a modification appliedaccording to the various aspects of the present invention, the inventionis by no means limited to this characteristic. Rather any characteristicor combination of characteristics may be applied. Another example of acharacteristic of a frame that may be used to affect the modification tobe applied comprises noise type (such as spatial noise, row noise, pixelnoise, temporal noise, or combinations of these, for example).

As used herein, whether in the above description or the followingclaims, the terms “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” and the like are to be understood to beopen-ended, that is, to mean including but not limited to. Also, itshould be understood that the terms “about,” “substantially,” and liketerms used herein when referring to a dimension or characteristic of acomponent indicate that the described dimension/characteristic is not astrict boundary or parameter and does not exclude variations therefromthat are functionally similar. At a minimum, such references thatinclude a numerical parameter would include variations that, usingmathematical and industrial principles accepted in the art (e.g.,rounding, measurement or other systematic errors, manufacturingtolerances, etc.), would not vary the least significant digit.

Any use of ordinal terms such as “first,” “second,” “third,” etc., inthe following claims to modify a claim element does not by itselfconnote any priority, precedence, or order of one claim element overanother, or the temporal order in which acts of a method are performed.Rather, unless specifically stated otherwise, such ordinal terms areused merely as labels to distinguish one claim element having a certainname from another element having a same name (but for use of the ordinalterm).

The term “each” may be used in the following claims for convenience indescribing characteristics or features of multiple elements, and anysuch use of the term “each” is in the inclusive sense unlessspecifically stated otherwise. For example, if a claim defines two ormore elements as “each” having a characteristic or feature, the use ofthe term “each” is not intended to exclude from the claim scope asituation having a third one of the elements which does not have thedefined characteristic or feature.

The above described preferred embodiments are intended to illustrate theprinciples of the invention, but not to limit the scope of theinvention. Various other embodiments and modifications to thesepreferred embodiments may be made by those skilled in the art withoutdeparting from the scope of the present invention. For example, in someinstances, one or more features disclosed in connection with oneembodiment can be used alone or in combination with one or more featuresof one or more other embodiments. More generally, the various featuresdescribed herein may be used in any working combination.

1. A method of processing a frame in a digital video signal, the methodincluding: (a) receiving a first digital video signal having frames at afirst frame rate; (b) for a respective frame of the first digital videosignal, replicating the respective frame multiple times to produce aseries of replicated frames, each replicated frame defined at least inpart by respective pixel data specifying each pixel included in therespective replicated frame; (c) for at least some of the respectivepixels for at least one of the replicated frames, randomly modifying therespective pixel data, the random modification of pixel data for arespective pixel resulting in a modified replicated frame; and (d)including the modified replicated frame in a second digital video signalproduced from additional replicated frames of the first digital videosignal so as to have a second frame rate greater than the first framerate.
 2. The method of claim 1 further including: (a) detecting a noiselevel in the first digital video signal; (b) selecting a modificationlevel for the random modification of the respective pixel data based onthe detected noise level; and (c) wherein the random modification of therespective pixel data is performed according to the selectedmodification level.
 3. The method of claim 1 further including: (a)detecting a noise level specific to a first color component of the firstdigital video signal; (b) selecting a first color component modificationlevel for the random modification of the respective pixel data based onthe detected noise level; and (c) wherein the random modification of therespective pixel data is applied to a first color component of the pixeldata according to the selected first color component modification level,and the random modification of the respective pixel data is applied tocolor components other than the first color component according to asecond modification level different from the first color componentmodification level.
 4. The method of claim 1 wherein the modification ofthe respective pixel data is performed for each pixel in the replicatedframe.
 5. The method of claim 1 wherein the modification of therespective pixel data is performed for every nth pixel in a respectiveline of pixels for the replicated frame where n is an integer greaterthan
 1. 6. The method of claim 1 further including randomly selecting asubset of pixels in the at least one of the replicated frames andwherein the random modification of the respective pixel data isperformed for the randomly selected subset of pixels.
 7. The method ofclaim 1 further including: (a) detecting a noise level in a sub-area ofthe frames making up the first digital video signal; (b) selecting amodification level for the random modification of the respective pixeldata based on the detected noise level; and (c) wherein the randommodification of the respective pixel data is performed according to theselected modification level to a sub-area of the replicated framecorresponding to the sub-area of the frames making up the first digitalvideo signal.
 8. An apparatus for processing a frame of a digital videosignal, the apparatus including: (a) a frame replication deviceconfigured to receive a first digital video signal having a first framerate and to replicate a respective frame of the first digital signal toproduce a series of replicated frames, each replicated frame defined atleast in part by respective pixel data specifying each pixel for therespective replicated frame, and the series of replicated frames beingincluded in a second digital video signal having a second frame rategreater than the first frame rate; and (b) a frame modificationcontroller configured to, for at least some of the respective pixels inat least one of the replicated frames, randomly modify the respectivepixel data, the random modification of pixel data for a respective pixelof a respective replicated frame resulting in a modified replicatedframe included in the second digital video signal.
 9. The apparatus ofclaim 8 further including: (a) an image processing controller configuredto detect a noise level in the first digital video signal and to selecta modification level for the random modification of the respective pixeldata based on the detected noise level; and (b) wherein the framemodification controller is configured to modify the respective pixeldata according to the selected modification level.
 10. The apparatus ofclaim 8 further including: (a) an image processing controller configuredto detect a noise level specific to a first color component of the firstdigital video signal and to select a first color component modificationlevel for the random modification of the respective pixel data based onthe detected noise level; and (b) wherein the frame modificationcontroller is configured to modify a first color component of therespective pixel data according to the selected first color componentmodification level, and to modify color components of the respectivepixel data other than the first color component according to a secondmodification level different from the first color component modificationlevel.
 11. The apparatus of claim 8 wherein the frame modificationcontroller is configured to modify the pixel data for each pixel in therespective replicated frame.
 12. The apparatus of claim 8 wherein theframe modification controller is configured to modify the pixel data forevery nth pixel in a respective line of pixels for the replicated framewhere n is an integer greater than
 1. 13. The apparatus of claim 8wherein the frame modification controller is configured to randomlyselect a subset of pixels in the at least one of the replicated framesand wherein the random modification of the respective pixel data isperformed for the randomly selected subset of pixels.
 14. The apparatusof claim 8 further including: (a) an image processing controllerconfigured to detect a noise level in a sub-area of the frames making upthe first digital video signal and to select a modification level forthe random modification of the respective pixel data based on thedetected noise level; and (b) wherein the frame modification controlleris configured to apply the selected modification level to modify therespective pixel data of pixels in a sub-area of the replicated framecorresponding to the sub-area of the frames making up the first digitalvideo signal.
 15. A video system including: (a) an image sensor assemblyconfigured to generate a first digital video signal having a first framerate; (b) a camera control module configured to receive the firstdigital video signal and to provide control signals for the image sensorassembly; (c) a video frame replication device included in the cameracontrol module, the video frame replication device configured to producea series of replicated frames for a respective frame of the firstdigital video signal, each replicated frame defined at least in part byrespective pixel data specifying each pixel included for the respectivereplicated frame, and the series of replicated frames being included ina second digital video signal having a second frame rate greater thanthe first frame rate; and (d) a frame modification controller includedin the camera control module, the frame modification controllerconfigured to, for at least some of the pixel data defining pixels in atleast one of the replicated frames, randomly modify the respective pixeldata, the random modification of pixel data resulting in a modifiedreplicated frame included in the second digital video signal.
 16. Thesystem of claim 15 further including: (a) an image processing controllerconfigured to detect a noise level in the first digital video signal andto select a modification level for the random modification of therespective pixel data based on the detected noise level; and (b) whereinthe frame modification controller is configured to modify the respectivepixel data according to the selected modification level.
 17. The systemof claim 15 further including: (a) an image processing controllerconfigured to detect a noise level specific to a first color componentof the first digital video signal and to select a first color componentmodification level for the random modification of the respective pixeldata based on the detected noise level; and (b) wherein the framemodification controller is configured to modify a first color componentof the respective pixel data according to the selected first colorcomponent modification level, and to modify color components of therespective pixel data other than the first color component according toa second modification level different from the first color componentmodification level.
 18. The system of claim 15 wherein the framemodification controller is configured to modify the pixel data for eachpixel in the respective replicated frame.
 19. The system of claim 15wherein the frame modification controller is configured to modify thepixel data for every nth pixel in a respective line of pixels for thereplicated frame where n is an integer greater than
 1. 20. The system ofclaim 15 wherein the frame modification controller is configured torandomly select a subset of pixels in the at least one of the replicatedframes and wherein the random modification of the respective pixel datais performed for the randomly selected subset of pixels.